-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CDK for .NET cannot handle Swedish characters in TEMP/TMP path #7456
Comments
When looking at this again, I wonder if its a JSII + Windows issue and not specific to those characters. @RomainMuller does this look familiar relating to recent windows fixes in JSII? |
I suspect that it's some kind of encoding/decoding issue and that the problem is generally related to non-ascii characters? Btw, I see that the exception I wrote in the issue was when the TMP was set to my windows profile folder name. Let me know if you need more information to reproduce the issue or if you want me to provide more information |
Sure as hell looks like a decoding error. Some Unicode got garbled by someone somewhere! I'm half wondering it this is because the file system's code page isn't UTF-8 but we attempt a UTF8-encoded path... I hope I'm wrong. |
I can already comment that I was not able to reproduce this issue on macOS: $ mkdir -p .tmp/path_with_å_in_it
$ export TMPDIR=${PWD}/.tmp/path_with_å_in_it
$ cdk synth # in a .NET App directory I can see temporary directories (including that containing the |
@pergardebrink in order to get a better picture of what is happening in your runtime, would you be able to provide us with the console output generated when running your app with the I think you're correct that there's an encoding mishap somewhere, but I don't know if it happens within the .NET process, in the JS process, or during information exchange between them... The trace hopefully will help clarify that... |
Sure!
I get this when running cdk synth:
|
Okay so it looks like the garbled path is actually coming from .NET: {
"api": "load",
"name": "@aws-cdk/cloud-assembly-schema",
"version": "1.42.0",
"tarball": "c:\\temp\\path_with_´┐¢_in_it\\missn253.l5x\\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.42.0.tgz"
} That narrows the problem space a bit :) |
I'm curious what the following outputs in your environment @pergardebrink: using System;
using System.Text;
public class Program
{
public static void Main()
{
Console.WriteLine($"path_with_å_in_it ({Console.OutputEncoding})");
Console.OutputEncoding = Encoding.UTF8;
Console.WriteLine($"path_with_å_in_it ({Console.OutputEncoding})");
}
} On my environments, it outputs: path_with_å_in_it (System.Text.UTF8Encoding)
path_with_å_in_it (System.Text.UTF8Encoding+UTF8EncodingSealed) This looks to behave correctly for me (i.e: the I'm asking this because it appears |
I've prepared a possible fix for this problem... I'm not too sure how I can test that it actually solves the problem, short of actually having you try that. Thankfully, |
the output from your code snippet on my machine becomes (targeting netcoreapp3.1):
Yes, I'd be happy to test if you can give me a pre-release package! |
I'd guess I'll post the prerelease bundle here once it's available... |
Here's prerelease builds of the NuGet packages for .NET Runtime: 1.10.1-pr1913.0.zip, PGP signature from yours truly: 1.10.1-pr1913.0.zip.txt. |
Yes! They worked perfect, so you got the correct fix for it! |
The following code:
Gives me this:
But I guess the problem is the same with this codepage? Regardless, your fix solves the problem I'm having! Let me know if there's anything else you want me to verify if so. Other than that, thanks you so much for the help in investigating the issue! How far away will it be before it can be in a released version (not that it's super urgent)? |
Oh that's the other Codepage that could have been, but for some reason I though it couldn't be this one. But yeah whatever - they're both codepages that have representations for those fancy characters that are different from UTF-8's, which is our problem. Glad the prerelease build worked. Now just have to wait for a colleague to get around to review the PR :) |
In certain environments, the temporary directory path may contain unicode characters. If those are not properly encoded as UTF-8 when sent to the `@jsii/kernel` process, a failure will occur when trying to access such paths. This change contains fixes for two possible sources of problems in such situations: 1. In `@jsii/runtime`, the `SyncStdio` class was a little too eager in turning it's `Buffer` to UTF-8 strings, as it could have only a part of a multi-byte character sequence (the conversion would then fail or result in corruption). Instead, it now looks for `\n` directly on the `Buffer`, and only performs string conversion once one has been found. 2. In the .NET Runtime, the `NodeProcess` class would spawn using the `System.Diagnostic.Process` class without specifying input and output encodings, which by default are the `System.Console` encoding. If that happens to not be `UTF-8`, the result could be unreliable. Instead, the encodings are now forced to `System.Text.Encoding.UTF8` for all three pipes of the child process (`Input`, `Output` and `Error`). Reproducing the conditions for the failure reported in aws/aws-cdk#7456 is somewhat difficult, especially in the context of CI/CD. This makes it difficult to validate those fixes actually deliver on their promises.
This issue with the TMP folder seems to be gone now right in the latest version i use now, but I wanted to check something else before I open a new issue as I've probably discovered a similar issue (that might have a similar fix?): I bootstrapped like this: #3463 (comment) and set the
Let me know if you want me to join your team as a living integration test ;) I'm not sure it's the same case as I currently don't know how to workaround the issue to verify that as I haven't investigated where it got the string "aws-cdk-PerGårdebrink" from (it seems the regex considers the non ascii characters not valid) cdk version output:
|
I can confirm now that the username seem to be the issue here. I impersonated another local windows user on my machine using the same AWS credentials and now I can deploy the stack above! I see that the aws-cdk-PerGårdebrink is constructed here:
|
I am having this issue with a french "é" character in my name with CDK 1.80.0 / .NET 3.1.403
Any workaround? |
@cAlebcore my workaround was to set the TMP environment variable to be somewhere else that didn't contain any special characters! |
This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled. |
CDK for .NET Core does not seem to be able to handle paths with non-ascii characters in them. My username in Windows is having an
å
character, which seems to cause an issue becuase the TEMP folder contains that letter in it's path.Reproduction Steps
Error Log
An exception is thrown:
Unhandled exception. Amazon.JSII.Runtime.JsiiException: ENOENT: no such file or directory, stat 'C:\Users\PerG´┐¢rdebrink\AppData\Local\Temp\x0c4glzt.kiq\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.33.1.tgz'
Environment
Other
This is 🐛 Bug Report
The text was updated successfully, but these errors were encountered: