stack and ghc don't work well on Windows when the user name has non-ASCII chars in it

[nh2]

This is a meta-issue collecting all information about stack, and tools stack calls, failing in the case that you have a non-ASCII char in your user name on Windows.

For example, if you name your Windows user 日 so your home dir, including some stack related files, will be under C:\Users\日.

The goal for this issue to be closed is that you can just install stack and compile small and large big Haskell projects successfully without roadblocks, even when your user is named that way.

In general, there are two settings on Windows that affect the encoding behaviour:

• The global system locale setting (as can be set ​this way)
• The per-terminal/process code page (as can be seen/set using chcp in cmd.exe or PowerShell)

Extra problems in the usage of stack arise when you have are using paths with non-unicode file names, but have set the system locale set to English (United states), like shown here:

As per my current understanding, if programmed correctly, applications can be made to work no matter what your locale, code page, and user name is. When something doesn't work (as shown in various cases below), it usually indicates a bug in stack or (in most cases I've encountered so far) one of the upstream programs that it calls.

We'd like to fix all of those, in stack or upstreams, so that

• developers who happen to have non-ASCII user names names don't have a harder time working with Haskell
• developers can efficiently test whether their compiled projects work fine for non-ASCII users, by being able to run the change-compile-run workflow under a user with non-ASCII chars in its name

---

Maintained list of known issues:

• GHC bug: ghc-pkg list crashes on Windows when unicode character is in the path
  • This happens with some combinations of system locale and code page (details in the bug) when stack build calls out to ghc-pkg list.
• stack setup failing with commitBuffer: invalid argument (invalid character)
  • That issue is closed and a documented workaround is to set the system locale English (US)` but that's not good enough because other things fail when that is done.
• stack fails unpacking ghc on Windows under non-ASCII path because 7zip version is too old
• GHC bug: GHC fails to execute gcc on Windows when unicode character is in the path
  • This happens at the end of stack setup when stack tries to compile the stack-sanity-check file.

[nh2]

The kind people on the haskell-jp Slack let me know that many of them change their user names to ASCII chars in order to work around problems as covered by this issue.

Let's make that a thing of the past!

[nh2]

Found another issue: https://ghc.haskell.org/trac/ghc/ticket/15096

...
GHC installed to C:\Users\?\AppData\Local\Programs\stack\x86_64-windows\ghc-8.0.2\
The GHC located at C:\Users\µùÑ\AppData\Local\Programs\stack\x86_64-windows\ghc-8.0.2\bin\ghc.EXE failed to compile a sa
nity check. Please see:

    http://docs.haskellstack.org/en/stable/install_and_upgrade/

for more information. Exception was:
Running C:\Users\µùÑ\AppData\Local\Programs\stack\x86_64-windows\ghc-8.0.2\bin\ghc.EXE C:\Users\µùÑ\AppData\Local\Temp\2
\stack-sanity-check22080\Main.hs -no-user-package-db in directory C:\Users\µùÑ\AppData\Local\Temp\2\stack-sanity-check22
080\ exited with ExitFailure 1

[1 of 1] Compiling Main             ( C:\Users\?\AppData\Local\Temp\2\stack-sanity-check22080\Main.hs, C:\Users\?\AppDat
a\Local\Temp\2\stack-sanity-check22080\Main.o )


<no location info>: error:
    Warning: Couldn't figure out C compiler information!
             Make sure you're using GNU gcc, or clang
Unable to start C:\Users\?\AppData\Local\Programs\stack\x86_64-windows\ghc-8.0.2\mingw\bin/realgcc.exe (error code: 123)
`gcc.exe' failed in phase `Assembler'. (Exit code: 1)

For the full invocation see #3987 (comment).

[koflerdavid]

Here is another: #3976

[BjoernLoetters]

I just ran into this problem (as my name and windows username contains the letter ö) and found this issue. More than 4 years have passed and it does not seem like this issue is resolved. Is there any progress regarding the support of unicode paths in stack? It's extremely frustating for people like me, that this is still an open issue in 2022.

Steps to reproduce:

1. Install a fresh new instance of stack
2. Run stack setup

The GHC located at C:\Users\Björn\AppData\Local\Programs\stack\x86_64-windows\ghc-9.0.2\bin\ghc-9.0.2.exe failed to compile a sanity check. Please see:

    http://docs.haskellstack.org/en/stable/install_and_upgrade/

for more information. Exception was:
Received ExitFailure 1 when running
Raw command: "C:\\Users\\Bj\246rn\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.0.2\\bin\\ghc-9.0.2.exe" "C:\\Users\\Bj\246rn\\AppData\\Local\\Temp\\stack-sanity-check-44814962d8d4b8ac\\Main.hs" -no-user-package-db
Standard output:
[1 of 1] Compiling Main             ( C:\Users\Bjstack.exe: <stderr>: commitAndReleaseBuffer: invalid argument (invalid character)

Changing the code page of the shell to 65001 (UTF-8) does not fix the issue but triggers another exception:

The GHC located at C:\Users\Björn\AppData\Local\Programs\stack\x86_64-windows\ghc-9.0.2\bin\ghc-9.0.2.exe failed to compile a sanity check. Please see:

    http://docs.haskellstack.org/en/stable/install_and_upgrade/

for more information. Exception was:
Received ExitFailure 1 when running
Raw command: "C:\\Users\\Bj\246rn\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.0.2\\bin\\ghc-9.0.2.exe" "C:\\Users\\Bj\246rn\\AppData\\Local\\Temp\\stack-sanity-check-60af86a6df69b8ac\\Main.hs" -no-user-package-db
Run from: C:\Users\Björn\AppData\Local\Temp\stack-sanity-check-60af86a6df69b8ac\
Standard output:

[1 of 1] Compiling Main             ( C:\Users\Björn\AppData\Local\Temp\stack-sanity-check-60af86a6df69b8ac\Main.hs, C:\Users\Björn\AppData\Local\Temp\stack-sanity-check-60af86a6df69b8ac\Main.o )
Linking C:\Users\Björn\AppData\Local\Temp\stack-sanity-check-60af86a6df69b8ac\Main.exe ...
Standard error:


<no location info>: error:
    Warning: Couldn't figure out C compiler information!
             Make sure you're using GNU gcc, or clang
`windres.exe' failed in phase `Windres'. (Exit code: 1)
ghc-9.0.2.exe: fd:5: hGetContents: invalid argument (invalid byte sequence)

Maybe this issue is also related to ghc and/or the way how stack invokes ghc?

[hasufell]

The way forward is simply using the new filepath types: https://hasufell.github.io/posts/2022-06-29-fixing-haskell-filepaths.html

[mpilgrem]

@kuchenkruste, thanks for reporting this. I'm a Windows user myself, so I want Stack to support fully Windows users (which is one of its stated goals). To help me understand better your circumstances, can you tell me your Windows version, the terminal software you are using, and the version of Stack (stack --version).

[mpilgrem]

I tied this experiment with Stack (which is not @kuchenkruste's problem, that problem is the path to the Stack-supplied GHC):

I changed my Stack root to C:\test\björn\sr:

$Env:STACK_ROOT="C:\test\björn\sr"

I commanded stack new my-project and cd new-project and stack build. It fails with (I've reformatted the output to make it clearer):

[1 of 2] Compiling Main             ( C:\test\björn\sr\setup-exe-src\setup-Z6RU0evB.hs, C:\test\björn\sr\setup-exe-src\setup-Z6RU0evB.o )
Assembler messages:
Fatal error: can't create C:\\test\\björn\\sr\\setup-exe-src\\setup-Z6RU0evB.o.tmp: No such file or directory
`gcc.exe' failed in phase `Assembler'. (Exit code: 1)

--  While building simple Setup.hs (scroll up to its section to see the error) using:
      C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.0.2\bin\ghc-9.0.2.exe
        -rtsopts
        -threaded
        -clear-package-db
        -global-package-db
        -hide-all-packages
        -package base
        -main-is StackSetupShim.mainOverride
        -package Cabal-3.4.1.0
        C:\test\björn\sr\setup-exe-src\setup-Z6RU0evB.hs
        C:\test\björn\sr\setup-exe-src\setup-shim-Z6RU0evB.hs
        -o C:\test\björn\sr\setup-exe-cache\x86_64-windows\tmp-Cabal-simple_Z6RU0evB_3.4.1.0_ghc-9.0.2.exe
    Process exited with code: ExitFailure 1

GCC is GHC-supplied:

stack exec -- which gcc
/c/Users/mike/AppData/Local/Programs/stack/x86_64-windows/ghc-9.0.2/mingw/bin/gcc

The version of GCC is:

stack exec -- gcc --version
realgcc.exe (Rev3, Built by MSYS2 project) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE

My immediate reaction (I may be wrong) is that seems to be an upstream problem with the MSYS2-supplied gcc.exe. Some of my searches suggests that Cygwin lacks Unicode support (see, for example, msys2/MINGW-packages#7188 - which is from as recently as October 2020). I understand that some parts of MSYS2 build upon Cygwin (see, for example, https://www.msys2.org/wiki/How-does-MSYS2-differ-from-Cygwin/).

I think the sorry answer is that Stack's ability to support 'björn' (etc) in some paths is currently constrained by the inability of the tools on which Stack relies to support it.

[mpilgrem]

Based on my experiments, @kuchenkruste, the 'raw' command:

C:\\Users\\Bj\246rn\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.0.2\\bin\\ghc-9.0.2.exe"
    "C:\\Users\\Bj\246rn\\AppData\\Local\\Temp\\stack-sanity-check-44814962d8d4b8ac\\Main.hs"
    -no-user-package-db

would fail because the GHC-supplied GCC can't handle the path to Main.hs.

I think the work-around is to use Stack's global YAML configuration (config.yaml) to set local-programs-path to be a path that the GHC-supplied GCC can tolerate. See https://docs.haskellstack.org/en/stable/yaml_configuration/#local-programs-path.

[mpilgrem]

I should add (1) I am using Windows Terminal on Windows 11 (with a UK/English set up) and (2) that my experiments included using GHC directly, with:

C:/Users/mike/AppData/Local/Programs/stack/x86_64-windows/ghc-9.0.2/bin/ghc c:\test\björn\Main.hs

This fails with:

[1 of 1] Compiling Main             ( C:\\test\björn\Main.hs, C:\\test\björn\Main.o )
Assembler messages:
Fatal error: can't create C:\\\\test\\björn\\Main.o.tmp: No such file or directory
`gcc.exe' failed in phase `Assembler'. (Exit code: 1)

[mpilgrem]

There is also GHC issue #12971 'Paths are encoded incorrectly when invoking GCC' from five years ago, where @Mistuke identified the problem as being 'upstream'.

[mpilgrem]

I am closing this issue because repeating my experiment above, things now work as intended.