Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDK for .NET cannot handle Swedish characters in TEMP/TMP path #7456

Closed
pergardebrink opened this issue Apr 20, 2020 · 20 comments
Closed

CDK for .NET cannot handle Swedish characters in TEMP/TMP path #7456

pergardebrink opened this issue Apr 20, 2020 · 20 comments
Labels
bug This issue is a bug. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. effort/small Small work item – less than a day of effort jsii This issue originates in jsii, or this feature must be implemented in jsii. language/dotnet Related to .NET bindings p1

Comments

@pergardebrink
Copy link

pergardebrink commented Apr 20, 2020

CDK for .NET Core does not seem to be able to handle paths with non-ascii characters in them. My username in Windows is having an å character, which seems to cause an issue becuase the TEMP folder contains that letter in it's path.

Reproduction Steps

  • SET TMP to c:\path_with_å_in_it
  • Run cdk synthesize, where you have a csharp project (or any cdk command it seems)

Error Log

An exception is thrown:

Unhandled exception. Amazon.JSII.Runtime.JsiiException: ENOENT: no such file or directory, stat 'C:\Users\PerG´┐¢rdebrink\AppData\Local\Temp\x0c4glzt.kiq\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.33.1.tgz'

Environment

  • CLI Version : 1.33.1
  • Framework Version: .NET Core 3.1
  • OS : Windows 10 (1909)
  • Language : Swedish

Other

Unhandled exception. Amazon.JSII.Runtime.JsiiException: ENOENT: no such file or directory, stat 'C:\Users\PerG´┐¢rdebrink\AppData\Local\Temp\x0c4glzt.kiq\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.33.1.tgz'
   at Amazon.JSII.Runtime.Services.Client.TryDeserialize[TResponse](String responseJson)
   at Amazon.JSII.Runtime.Services.Client.ReceiveResponse[TResponse]()
   at Amazon.JSII.Runtime.Services.Client.Send[TRequest,TResponse](TRequest requestObject)
   at Amazon.JSII.Runtime.Services.Client.Load(LoadRequest request)
   at Amazon.JSII.Runtime.Services.Client.Load(String name, String version, String tarball)
   at Amazon.JSII.Runtime.Services.Client.LoadPackage(String package, String version, String tarballPath)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase..ctor(Type nativeType, String fullyQualifiedName)
   at Amazon.JSII.Runtime.Deputy.JsiiClassAttribute..ctor(Type nativeType, String fullyQualifiedName, String parametersJson)
   at System.Reflection.CustomAttribute._CreateCaObject(RuntimeModule pModule, RuntimeType type, IRuntimeMethodInfo pCtor, Byte** ppBlob, Byte* pEndBlob, Int32* pcNamedArgs)
   at System.Reflection.CustomAttribute.CreateCaObject(RuntimeModule module, RuntimeType type, IRuntimeMethodInfo ctor, IntPtr& blob, IntPtr blobEnd, Int32& namedArgs)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeType type, RuntimeType caType, Boolean inherit)
   at System.RuntimeType.GetCustomAttributes(Type attributeType, Boolean inherit)
   at System.Attribute.GetCustomAttributes(MemberInfo element, Type type, Boolean inherit)
   at System.Attribute.GetCustomAttribute(MemberInfo element, Type attributeType, Boolean inherit)
   at System.Reflection.CustomAttributeExtensions.GetCustomAttribute[T](MemberInfo element)
   at Amazon.JSII.Runtime.ReflectionUtils.GetClassAttribute(Type type)
   at Amazon.JSII.Runtime.Deputy.DeputyBase..ctor(DeputyProps props)
   at Constructs.Construct..ctor(DeputyProps props)
   at Amazon.CDK.Construct..ctor(DeputyProps props)
   at Amazon.CDK.App..ctor(IAppProps props)
   at Temp.Program.Main(String[] args) in C:\git\temp\src\Temp\Program.cs:line 12
Subprocess exited with error 3762504530

This is 🐛 Bug Report

@pergardebrink pergardebrink added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Apr 20, 2020
@SomayaB SomayaB added jsii This issue originates in jsii, or this feature must be implemented in jsii. language/dotnet Related to .NET bindings labels Apr 20, 2020
@MrArnoldPalmer MrArnoldPalmer added p1 and removed needs-triage This issue or PR still needs to be triaged. labels May 4, 2020
@MrArnoldPalmer MrArnoldPalmer added the effort/small Small work item – less than a day of effort label Aug 17, 2020
@MrArnoldPalmer
Copy link
Contributor

When looking at this again, I wonder if its a JSII + Windows issue and not specific to those characters. @RomainMuller does this look familiar relating to recent windows fixes in JSII?

@pergardebrink
Copy link
Author

When looking at this again, I wonder if its a JSII + Windows issue and not specific to those characters. @RomainMuller does this look familiar relating to recent windows fixes in JSII?

I suspect that it's some kind of encoding/decoding issue and that the problem is generally related to non-ascii characters?

Btw, I see that the exception I wrote in the issue was when the TMP was set to my windows profile folder name.
If you set TMP to the path in the step above, the exception message would of course be something like Unhandled exception. Amazon.JSII.Runtime.JsiiException: ENOENT: no such file or directory, stat 'C:\path_with_´┐¢in_it\x0c4glzt.kiq\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.33.1.tgz'

Let me know if you need more information to reproduce the issue or if you want me to provide more information

@RomainMuller
Copy link
Contributor

RomainMuller commented Aug 17, 2020

Sure as hell looks like a decoding error. Some Unicode got garbled by someone somewhere!

I'm half wondering it this is because the file system's code page isn't UTF-8 but we attempt a UTF8-encoded path... I hope I'm wrong.

@RomainMuller
Copy link
Contributor

I can already comment that I was not able to reproduce this issue on macOS:

$ mkdir -p .tmp/path_with_å_in_it
$ export TMPDIR=${PWD}/.tmp/path_with_å_in_it
$ cdk synth # in a .NET App directory

I can see temporary directories (including that containing the jsii-runtime.js and related items, as well as the tarballs, etc...) created under my .tmp/path_with_å_in_it directory and my app synthesizes correctly.

@RomainMuller
Copy link
Contributor

@pergardebrink in order to get a better picture of what is happening in your runtime, would you be able to provide us with the console output generated when running your app with the JSII_DEBUG=1 environment variable set?

I think you're correct that there's an encoding mishap somewhere, but I don't know if it happens within the .NET process, in the JS process, or during information exchange between them... The trace hopefully will help clarify that...

@pergardebrink
Copy link
Author

Sure!

mkdir c:\temp\path_with_å_in_it
SET TMP=c:\temp\path_with_å_in_it
SET JSII_DEBUG=1

I get this when running cdk synth:

> {"api":"load","name":"@aws-cdk/cloud-assembly-schema","version":"1.42.0","tarball":"c:\\temp\\path_with_´┐¢_in_it\\missn253.l5x\\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.42.0.tgz"}
[@jsii/kernel] load {
  api: 'load',
  name: '@aws-cdk/cloud-assembly-schema',
  version: '1.42.0',
  tarball: 'c:\\temp\\path_with_´┐¢_in_it\\missn253.l5x\\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.42.0.tgz'
}
[@jsii/kernel] creating jsii-kernel modules workdir: C:\Users\PERGRD~1\AppData\Local\Temp\jsii-kernel-jZqBoR
[@jsii/kernel] removing staging directory: C:\Users\PERGRD~1\AppData\Local\Temp\jsii-kernel-install-staging-LElxzI
< {"error":"ENOENT: no such file or directory, stat 'c:\\temp\\path_with_´┐¢_in_it\\missn253.l5x\\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.42.0.tgz'","stack":"Error: ENOENT: no such file or directory, stat 'c:\\temp\\path_with_´┐¢_in_it\\missn253.l5x\\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.42.0.tgz'\n    at Object.statSync (fs.js:932:3)\n    at extractFileSync (c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:12640:19)\n    at Object.module.exports (c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:12607:33)\n    at Kernel.load (c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:7576:17)\n    at KernelHost.processRequest (c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:7388:28)\n    at KernelHost.run (c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:7328:14)\n    at Object.<anonymous> (c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:7296:6)\n    at __webpack_require__ (c:\\temp\\path_with_├Ñ_in_it\Unhandled exception. \bfex4n1e.2vl\\jsii-runtime.js:20:30)\n    at c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:84:18\n    at Object.<anonymous> (c:\\temp\\path_with_├Ñ_in_it\\bfex4n1e.2vl\\jsii-runtime.js:87:10)"}
Amazon.JSII.Runtime.JsiiException: ENOENT: no such file or directory, stat 'c:\temp\path_with_´┐¢_in_it\missn253.l5x\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.42.0.tgz'
   at Amazon.JSII.Runtime.Services.Client.TryDeserialize[TResponse](String responseJson)
   at Amazon.JSII.Runtime.Services.Client.ReceiveResponse[TResponse]()
   at Amazon.JSII.Runtime.Services.Client.Send[TRequest,TResponse](TRequest requestObject)
   at Amazon.JSII.Runtime.Services.Client.Load(LoadRequest request)
   at Amazon.JSII.Runtime.Services.Client.Load(String name, String version, String tarball)
   at Amazon.JSII.Runtime.Services.Client.LoadPackage(String package, String version, String tarballPath)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.DeputyBase..ctor(DeputyProps props)
   at Constructs.Construct..ctor(DeputyProps props)
   at Amazon.CDK.Construct..ctor(DeputyProps props)
   at Amazon.CDK.App..ctor(IAppProps props)
   at Cdk.Program.Main(String[] args) in C:\git\cdktest\src\cdk\Program.cs:line 6
Subprocess exited with error 3762504530

@RomainMuller
Copy link
Contributor

Okay so it looks like the garbled path is actually coming from .NET:

{
  "api": "load",
  "name": "@aws-cdk/cloud-assembly-schema",
  "version": "1.42.0",
  "tarball": "c:\\temp\\path_with_´┐¢_in_it\\missn253.l5x\\Amazon.CDK.CloudAssembly.Schema.aws-cdk-cloud-assembly-schema-1.42.0.tgz"
}

That narrows the problem space a bit :)

@RomainMuller
Copy link
Contributor

I'm curious what the following outputs in your environment @pergardebrink:

using System;
using System.Text;
					
public class Program
{
	public static void Main()
	{
		Console.WriteLine($"path_with_å_in_it ({Console.OutputEncoding})");
		Console.OutputEncoding = Encoding.UTF8;
		Console.WriteLine($"path_with_å_in_it ({Console.OutputEncoding})");
	}
}

On my environments, it outputs:

path_with_å_in_it (System.Text.UTF8Encoding)
path_with_å_in_it (System.Text.UTF8Encoding+UTF8EncodingSealed)

This looks to behave correctly for me (i.e: the å in the path does not seem to be a source of trouble).


I'm asking this because it appears System.Diagnostics.Process uses the Console encoding by default for the input/ouput streams...

@RomainMuller
Copy link
Contributor

I've prepared a possible fix for this problem... I'm not too sure how I can test that it actually solves the problem, short of actually having you try that. Thankfully, jsii's CI process produces prerelease packages that I can give you so you can try them.

@pergardebrink
Copy link
Author

the output from your code snippet on my machine becomes (targeting netcoreapp3.1):

path_with_å_in_it (System.Text.OSEncoding)
path_with_å_in_it (System.Text.UTF8Encoding+UTF8EncodingSealed)

Yes, I'd be happy to test if you can give me a pre-release package!

@RomainMuller
Copy link
Contributor

RomainMuller commented Aug 18, 2020

I'd guess System.Text.OSEncoding is the culprit here... Pretty sure for your culture this ends up being CodePage 20107. Well this confirms the fix I am proposing should work out.

I'll post the prerelease bundle here once it's available...

@RomainMuller
Copy link
Contributor

Here's prerelease builds of the NuGet packages for .NET Runtime: 1.10.1-pr1913.0.zip, PGP signature from yours truly: 1.10.1-pr1913.0.zip.txt.

@pergardebrink
Copy link
Author

Here's prerelease builds of the NuGet packages for .NET Runtime: 1.10.1-pr1913.0.zip, PGP signature from yours truly: 1.10.1-pr1913.0.zip.txt.

Yes! They worked perfect, so you got the correct fix for it!

@pergardebrink
Copy link
Author

I'd guess System.Text.OSEncoding is the culprit here... Pretty sure for your culture this ends up being CodePage 20107. Well this confirms the fix I am proposing should work out.

I'll post the prerelease bundle here once it's available...

The following code:

Console.WriteLine($"({Console.OutputEncoding}) Codepage: {Console.OutputEncoding.CodePage}");

Gives me this:

(System.Text.OSEncoding) Codepage: 850

But I guess the problem is the same with this codepage?

Regardless, your fix solves the problem I'm having! Let me know if there's anything else you want me to verify if so. Other than that, thanks you so much for the help in investigating the issue! How far away will it be before it can be in a released version (not that it's super urgent)?

@RomainMuller
Copy link
Contributor

Oh that's the other Codepage that could have been, but for some reason I though it couldn't be this one. But yeah whatever - they're both codepages that have representations for those fancy characters that are different from UTF-8's, which is our problem.

Glad the prerelease build worked. Now just have to wait for a colleague to get around to review the PR :)

RomainMuller added a commit to aws/jsii that referenced this issue Aug 18, 2020
In certain environments, the temporary directory path may contain
unicode characters. If those are not properly encoded as UTF-8 when sent
to the `@jsii/kernel` process, a failure will occur when trying to
access such paths.

This change contains fixes for two possible sources of problems in such
situations:

1. In `@jsii/runtime`, the `SyncStdio` class was a little too eager in
   turning it's `Buffer` to UTF-8 strings, as it could have only a part
   of a multi-byte character sequence (the conversion would then fail
   or result in corruption). Instead, it now looks for `\n` directly on
   the `Buffer`, and only performs string conversion once one has been
   found.
2. In the .NET Runtime, the `NodeProcess` class would spawn using the
   `System.Diagnostic.Process` class without specifying input and output
   encodings, which by default are the `System.Console` encoding. If
   that happens to not be `UTF-8`, the result could be unreliable.
   Instead, the encodings are now forced to `System.Text.Encoding.UTF8`
   for all three pipes of the child process (`Input`, `Output` and
   `Error`).

Reproducing the conditions for the failure reported in aws/aws-cdk#7456 is
somewhat difficult, especially in the context of CI/CD. This makes it difficult
to validate those fixes actually deliver on their promises.
@pergardebrink
Copy link
Author

pergardebrink commented Sep 9, 2020

This issue with the TMP folder seems to be gone now right in the latest version i use now, but I wanted to check something else before I open a new issue as I've probably discovered a similar issue (that might have a similar fix?):

I bootstrapped like this: #3463 (comment) and set the @aws-cdk/core:newStyleStackSynthesis": true in my cdk.json and after that, my deploy fails with the following message:

$ cdk deploy Pipeline-Staging --require-approval=never
Pipeline-Staging: deploying...

 ÔØî  Pipeline-Staging failed: Error: Could not assume role in target account (did you bootstrap the environment with the right '--trust's?): 1 validation error detected: Value 'aws-cdk-PerG├Ñrdebrink' at 'roleSessionName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\w+=,.@-]*
    at makeDetailedException (C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\aws-auth\sdk.ts:192:25)
    at C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\aws-auth\sdk.ts:164:19
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Function.lookup (C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\util\cloudformation.ts:26:24)
    at C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\util\cloudformation.ts:313:19
    at waitFor (C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\util\cloudformation.ts:188:20)
    at Function.lookup (C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\toolkit-info.ts:25:19)
    at CloudFormationDeployments.deployStack (C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\cloudformation-deployments.ts:151:25)
    at CdkToolkit.deploy (C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\lib\cdk-toolkit.ts:181:24)
    at main (C:\Users\PerGårdebrink\AppData\Roaming\npm\node_modules\aws-cdk\bin\cdk.ts:286:16) {
  message: "Could not assume role in target account (did you bootstrap the environment with the right '--trust's?): 1 validation error detected: Value 'aws-cdk-PerGårdebrink' at 'roleSessionName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*",
  code: 'CredentialsError',

Let me know if you want me to join your team as a living integration test ;)

I'm not sure it's the same case as I currently don't know how to workaround the issue to verify that as I haven't investigated where it got the string "aws-cdk-PerGårdebrink" from (it seems the regex considers the non ascii characters not valid)

cdk version output:

$ cdk --version
1.62.0 (build 8c2d7fc)

@pergardebrink
Copy link
Author

I can confirm now that the username seem to be the issue here. I impersonated another local windows user on my machine using the same AWS credentials and now I can deploy the stack above! I see that the aws-cdk-PerGårdebrink is constructed here:

RoleSessionName: `aws-cdk-${os.userInfo().username}`,

@cAlebcore
Copy link

cAlebcore commented Dec 27, 2020

I am having this issue with a french "é" character in my name with CDK 1.80.0 / .NET 3.1.403

cdk deploy
Unhandled exception. Amazon.JSII.Runtime.JsiiException: ENOENT: no such file or directory, stat 'C:\Users\Aur�lien\AppData\Local\Temp\3p5buo5x.kkb\Constructs.constructs-3.0.2.tgz'
   at Amazon.JSII.Runtime.Services.Client.TryDeserialize[TResponse](String responseJson)
   at Amazon.JSII.Runtime.Services.Client.ReceiveResponse[TResponse]()
   at Amazon.JSII.Runtime.Services.Client.Send[TRequest,TResponse](TRequest requestObject)
   at Amazon.JSII.Runtime.Services.Client.Load(LoadRequest request)
   at Amazon.JSII.Runtime.Services.Client.Load(String name, String version, String tarball)
   at Amazon.JSII.Runtime.Services.Client.LoadPackage(String package, String version, String tarballPath)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.JsiiTypeAttributeBase.Load(Assembly assembly)
   at Amazon.JSII.Runtime.Deputy.DeputyBase..ctor(DeputyProps props)
   at Constructs.Construct..ctor(DeputyProps props)
   at Amazon.CDK.Construct..ctor(DeputyProps props)
   at Amazon.CDK.Stage..ctor(DeputyProps props)
   at Amazon.CDK.App..ctor(IAppProps props)
   at Turbooo.Program.Main(String[] args) in C:\Users\Aurélien\source\repos\infrastructure\src\Program.cs:line 12
Subprocess exited with error 3762504530
cdk version
1.80.0 (build 31132ca)

Any workaround?

@pergardebrink
Copy link
Author

@cAlebcore my workaround was to set the TMP environment variable to be somewhere else that didn't contain any special characters!

@MrArnoldPalmer MrArnoldPalmer removed their assignment Jun 21, 2021
@github-actions
Copy link

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Jun 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. effort/small Small work item – less than a day of effort jsii This issue originates in jsii, or this feature must be implemented in jsii. language/dotnet Related to .NET bindings p1
Projects
None yet
Development

No branches or pull requests

5 participants