-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
target label paths with non-latin1 characters have inconsistent starlark semantics. #11602
Comments
This seems like a dup of #374. |
It is related to it, but it concerns a small detail - that reading the same file name on Windows and Linux can result in different strings. FIxing #374 would most likely fix this, but we could fix this discrepancy on it's own without doing all the work needed for the other issue. Thinking about it differently, 374 could possibly be a deep starlark change. We might be able to fix this problem at the the file system API layer. |
The fix to #374 requires changing the decoding applied as soon as input is obtained (from read and readdir) and the encoding applied immediately before strings are output (by print, FileWriteAction, Forge RPCs, and a few other places). It also requires changing the JVM-wide file.encoding property. It took me a week to prepare a CL to implement as an immediate breaking change, but it would be a project to put it behind a flag. |
I'm not sure where you are going with putting it behind a flag, but is your point that this can't be fixed without fixing #374 as well? If so, yes, this is a duplicate. |
On second thoughts, I'm surprised about the platform variation. The only parts that vary by platform in #374 are the functions used within glob to read the directory. On Unix we end up in NativePosixFiles.readdir; on Windows we end up (I assume) in java.io.WinNTFileSystem.list. It would be good to know what you get from the lowest levels of the awfully messy glob machinery when you read a directory entry named λ. Please do provide more details. |
I created what should be a repo here: https://github.com/tonyaiuto/bazel/tree/master/utf8_encode But.... I do not have access to a windows machine, so I can not be certain of it. It should match the pattern where I found the problem trying to write regression tests for ctx.actions.write(), which was why I filed this issue. I expect that on linux we will see bazel-bin/filename.txt
and on windows we will see
|
…UILD and .bzl input parsing. Internally BUILD and .bzl files are read as uninterpreted raw byte strings and not as UTF-8, so we should write content back out the same way. For example, if a BUILD file contains: ``` ... notice = "Copyright © 2019 Acme LLC", ``` Bazel will store the copyright symbol as two distinct octets ([0xc2, 0xa9]). When writing that with an action, the same two octets should be emitted. The behavior fixed by this change is to not interpret each character as a standalone Unicode code point to be encoded as UTF-8. While this change is appropriate for strings read from BUILD files, there is still a discrepancy in handling file paths with non Latin1 characters on Windows file systems. That is described in #11602. While this change does produce an observable behavior change, I am considering it a bug fix rather than an incompatible change because the previous behavior was both wrong and also (as described in #11602) inconsistent across different OSes. Closes #10174 RELNOTES: Fix behavior of ctx.actions.write so content is written without an incorrect encoding to UTF-8. See #10174 for details. PiperOrigin-RevId: 318056290
After further investigation: On Windows, Bazel's functions for reading directory entries (e.g. JavaIoFileSystem.getDirectoryEntries) trivially decodes the UTF-16 file names correctly to Java strings, which is inconsistent with the POSIX behavior in which we treat directory entries and file contents as Latin1. If such strings are later used as file names, all is well, because the reverse trivial encoding (UTF16->UTF-16) applies. Starlark code can however observe that the string elements are UTF-16 codes, not UTF-8 codes, and the lengths differ. By contrast, if such strings are later used as file contents, they will be incorrect, because the UTF-16 codes will be treated as UTF-8 codes and the lower-8 bits written out as bytes. Non-ASCII will become garbage. So, we have two choices. The first is to make getDirectoryEntries on Windows misinterpret file names just as it does in POSIX. I started this, but I no longer thing it is viable: we would need to do this in every place in the file system API that accepts one of Bazel's mis-encoded strings, otherwise, for example, glob() won't be able to see files with non-ASCII names because the mangled filenames it gets from the directory can't be statted. There are a large number of such places. (This is easy for POSIX, and already done, twice over in fact: once in the native file system wrappers by making explicit calls to JNI Latin1 functions instead of UTF8; and once in the JavaIoFileSystem when used on POSIX because of Java's existing -Dfile.encoding=latin1 mechanism. This mechanism doesn't exist on Windows because it isn't needed---filenames aren't bytes.) So this is a lot of work to introduce bugs to make Windows as bad as POSIX. The other choice is to fix the behavior on POSIX. That is issue #374. We know how to do it. It's an observable change to Starlark code and external compilers, and could thus be considered a breaking change, but I will argue that if a Starlark program or external compiler relies on anything more complex than that non-ASCII string literals and file names transit through Blaze, it is probably already subtly broken in many ways. I am not going to waste any more time on encoding issues unless it is a complete fix to #374. |
Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale. |
We need a way to turn issues into archive documents for the person who eventually wants to fix the unicode character handline |
@tetromino Is the plan to switch to representing strings as UTF-8 internally still alive? This seems separate from the much more complicated to solve #374, so maybe there is a way to get rid of the Latin-1 hack without also having to think about an escaping scheme for labels (#23335 (comment)). I started to work on a PR that maps between the internal Latin-1 representation and the external world on the filesystem level, similar to what @alandonovan described above: #23443 CC @meteorcloudy @tjgq for their input as they have been reviewing most of my Unicode fixes so far. |
IIRC, we do represent strings as UTF-8 internally, we just call everything Latin-1. The specific of this bug is that linux, windows, and macos all encode write non-Latin-1 characters in different bit patterns that are all UTF-8 compliant. Everything works as long as the host and exec platforms are the same. If you mix them, things can break. |
@fmeum - the plan is there, but delayed to probably next year (so not in Bazel 8.0). |
Everything works as long as the host and exec platforms are the same. If you mix them, things can break.
This is not true as far as I can tell. Remote execution correctly translates between the internal encoding and UTF-8 since
d91e05e, so mixing different host and exec platforms should actually not be a problem.
Subtle issues arise when Bazel uses the Java (N)IO APIs to interact with the filesystem, such as in JavaIoFileSystem or WindowsFileSystem. Java uses the encoding specified by `sun.jnu.encoding` for that and this can't be influenced reliably (it's always UTF-8 on macOS and can only be forced to Latin-1 on Linux if a latin-1 locale is installed, which isn't usually the case).
That's why I think that introducing a reencoding step to the filesystem layer would help squash the remaining issues. It's not too intrusive, there is already a central method for interop with Java APIs that we could extend.
|
If we have a source file with a non Latin1 character in the file name (e.g. λ.file)
Hilarity ensues while trying to figure out how to write these strings as part of an action.
Discovered while trying to fix #10174 More details to follow.
The text was updated successfully, but these errors were encountered: