[Error]stream did not contain valid UTF-8 #8009

sdbds · 2024-10-08T16:42:22Z

command: uv pip install crcmod==1.7
platform:windows
version:uv 0.4.19 (a451fb6 2024-10-07)

error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: crcmod==1.7
  Caused by: Failed to run `C:\Users\QINGLO~1\AppData\Local\Temp\.tmpSyOjmr\builds-v0\.tmps5aMOw\Scripts\python.exe`
  Caused by: stream did not contain valid UTF-8

The text was updated successfully, but these errors were encountered:

BurntSushi · 2024-10-08T16:49:01Z

Can you share the output of running chcp? That should print the codepage you're using.

konstin · 2024-10-08T16:51:59Z

Additionally, which terminal/shell are you using? I couldn't reproduce this on my windows machine (german locale, codepage 850, powershell)

Note crcmod==1.7 was published in 2010 and is only tested up to Python 3.1.

sdbds · 2024-10-08T17:02:32Z

Can you share the output of running chcp? That should print the codepage you're using.

Additionally, which terminal/shell are you using? I couldn't reproduce this on my windows machine (german locale, codepage 850, powershell)

Note crcmod==1.7 was published in 2010 and is only tested up to Python 3.1.

Chinese locale, codepage 936, powershell 7.4.5

sdbds · 2024-10-08T17:47:33Z

crcmod==1.7
# via oss2
oss2==2.19.0
# via funasr
funasr==1.1.5
# via fish-speech (./pyproject.toml)

kahojyun · 2024-10-09T02:20:03Z

Please try install with pip install -v --no-cache-dir crcmod==1.7 and uv pip install -vv -n crdmod==1.7. It should print the output of build script like:

❯ .venv\Scripts\pip.exe install --no-cache-dir -v crcmod==1.7
Using pip 24.2 from E:\tt\.venv\lib\site-packages\pip (python 3.8)
Collecting crcmod==1.7
  Downloading crcmod-1.7.tar.gz (89 kB)
  Running command python setup.py egg_info
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: crcmod
  Running command python setup.py bdist_wheel
  _crcfunext.c
     Creating library build\temp.win-amd64-cpython-38\Release\python3\src\_crcfunext.cp38-win_amd64.lib and object build\temp.win-amd64-cpython-38\Release\python3\src\_crcfunext.cp38-win_amd64.exp
  Generating code
  Finished generating code
  E:\tt\.venv\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
  !!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer or other
          standards-based tools.

          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************

  !!
    self.initialize_options()
  E:\tt\.venv\lib\site-packages\setuptools\command\bdist_wheel.py:115: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
    if get_flag("Py_DEBUG", hasattr(sys, "gettotalrefcount"), warn=(impl == "cp")):
  Building wheel for crcmod (setup.py) ... done
  Created wheel for crcmod: filename=crcmod-1.7-cp38-cp38-win_amd64.whl size=25415 sha256=777a3c9daadf2c05407249df3af462e8da226d1099a33f1f34328c1238e164e1
  Stored in directory: C:\Users\kahoj\AppData\Local\Temp\pip-ephem-wheel-cache-5cky5nzg\wheels\ca\5a\02\f3acf982a026f3319fb3e798a8dca2d48fafee7761788562e9
Successfully built crcmod
Installing collected packages: crcmod
Successfully installed crcmod-1.7

sdbds · 2024-10-09T08:30:45Z

Please try install with pip install -v --no-cache-dir crcmod==1.7 and uv pip install -vv -n crdmod==1.7. It should print the output of build script like:

❯ .venv\Scripts\pip.exe install --no-cache-dir -v crcmod==1.7
Using pip 24.2 from E:\tt\.venv\lib\site-packages\pip (python 3.8)
Collecting crcmod==1.7
  Downloading crcmod-1.7.tar.gz (89 kB)
  Running command python setup.py egg_info
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: crcmod
  Running command python setup.py bdist_wheel
  _crcfunext.c
     Creating library build\temp.win-amd64-cpython-38\Release\python3\src\_crcfunext.cp38-win_amd64.lib and object build\temp.win-amd64-cpython-38\Release\python3\src\_crcfunext.cp38-win_amd64.exp
  Generating code
  Finished generating code
  E:\tt\.venv\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
  !!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer or other
          standards-based tools.

          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************

  !!
    self.initialize_options()
  E:\tt\.venv\lib\site-packages\setuptools\command\bdist_wheel.py:115: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
    if get_flag("Py_DEBUG", hasattr(sys, "gettotalrefcount"), warn=(impl == "cp")):
  Building wheel for crcmod (setup.py) ... done
  Created wheel for crcmod: filename=crcmod-1.7-cp38-cp38-win_amd64.whl size=25415 sha256=777a3c9daadf2c05407249df3af462e8da226d1099a33f1f34328c1238e164e1
  Stored in directory: C:\Users\kahoj\AppData\Local\Temp\pip-ephem-wheel-cache-5cky5nzg\wheels\ca\5a\02\f3acf982a026f3319fb3e798a8dca2d48fafee7761788562e9
Successfully built crcmod
Installing collected packages: crcmod
Successfully installed crcmod-1.7

pip install -v --no-cache-dir crcmod==1.7
Using pip 23.3.1 from E:\Python310\lib\site-packages\pip (python 3.10)
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: crcmod==1.7 in e:\python310\lib\site-packages (1.7)

uv pip install -vv -n crdmod==1.7
    0.001468s DEBUG uv uv 0.4.19
 uv_requirements::specification::from_source source=crdmod==1.7
    0.003947s DEBUG uv_python::discovery Searching for default Python interpreter in system path or `py` launcher
    0.157329s DEBUG uv_python::discovery Found `cpython-3.10.11-windows-x86_64-none` at `E:\Code\fish-speech-for-windows\.venv\Scripts\python.exe` (active virtual environment)
    0.158000s DEBUG uv::commands::pip::operations Using Python 3.10.11 environment at .venv
    0.158378s DEBUG uv_fs Acquired lock for `.venv`
    0.170668s DEBUG uv::commands::pip::install At least one requirement is not satisfied: crdmod==1.7
 uv_client::linehaul::linehaul
    0.171926s DEBUG uv_client::base_client Using request timeout of 30s
 uv_resolver::flat_index::from_entries
 uv_resolver::resolver::solve
    0.173051s   0ms DEBUG uv_resolver::resolver Solving with installed Python version: 3.10.11
    0.173136s   0ms DEBUG uv_resolver::resolver Solving with target Python version: >=3.10.11
   uv_resolver::resolver::choose_version package=root
   uv_resolver::resolver::get_dependencies_forking package=root, version=0a0.dev0
     uv_resolver::resolver::get_dependencies package=root, version=0a0.dev0
    0.173560s   0ms DEBUG uv_resolver::resolver Adding direct dependency: crdmod==1.7
   uv_resolver::resolver::choose_version package=crdmod
 uv_resolver::resolver::process_request request=Versions crdmod
   uv_client::registry_client::simple_api package=crdmod
     uv_client::cached_client::get_cacheable
       uv_client::cached_client::read_and_parse_cache file=C:\Users\QINGLO~1\AppData\Local\Temp\.tmpSrw8eN\simple-v13\pypi\crdmod.rkyv
 uv_resolver::resolver::process_request request=Prefetch crdmod ==1.7
 uv_client::cached_client::from_path_sync path="C:\\Users\\QINGLO~1\\AppData\\Local\\Temp\\.tmpSrw8eN\\simple-v13\\pypi\\crdmod.rkyv"
        0.174370s   0ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/crdmod/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/crdmod/"
      0.342466s 168ms DEBUG uv_resolver::resolver Searching for a compatible version of crdmod (==1.7)
    0.342615s 169ms DEBUG uv_resolver::resolver No compatible version found for: crdmod
  × No solution found when resolving dependencies:
  ╰─▶ Because crdmod was not found in the package registry and you require crdmod==1.7, we can conclude that your
      requirements are unsatisfiable.
    0.343650s DEBUG uv_fs Released lock at `E:\Code\fish-speech-for-windows\.venv\.lock`

kahojyun · 2024-10-11T12:57:33Z

I think the non-UTF8 bytes come from the MSVC compiler output. It seems that the output of the compiler bypasses the PYTHONIOENCODING setting and writes directly to stdout/stderr using system default encoding.

## Summary Related issues: #8009 #7549 Although `PYTHONIOENCODING=utf-8` forces python to use UTF-8 for `stdout`/`stderr`, it can't prevent code like `sys.stdout.buffer.write()` or `subprocess.call(["cl.exe", ...])` to bypass the encoder. This PR uses lossy UTF-8 conversion to avoid decoding error. ## Alternative Using `bstr` crate might be better since it can preserve original information. Or we should follow the Windows convention, unset `PYTHONIOENCODING` and decode with system default encoding. ## Test Plan Running locally with non-ASCII character in `UV_CACHE_DIR` works fine, but I have no unit test plan. Testing locale problem is hard :(

BurntSushi · 2024-10-11T13:10:32Z

@sdbds With #8120 merged, when the next release comes out, could you let us know whether this issue is resolved or not? Thanks!

Super1Windcloud · 2024-10-11T15:15:26Z

I think the non-UTF8 bytes come from the MSVC compiler output. It seems that the output of the compiler bypasses the PYTHONIOENCODING setting and writes directly to stdout/stderr using system default encoding.我认为非 UTF8 字节来自 MSVC 编译器输出。编译器的输出似乎绕过了PYTHONIOENCODING设置并使用系统默认编码直接写入stdout / stderr 。

very nice , how do you find this reason ?

sdbds · 2024-10-12T11:12:51Z

@sdbds With #8120 merged, when the next release comes out, could you let us know whether this issue is resolved or not? Thanks!

I need to check lastest release version for this because my environment dont have rust...
wait for fixing version for python wheel.

konstin · 2024-10-12T18:36:09Z

I'm going to close this since #8120 is merged. Please report if the next release still doesn't work, we'll reopen and try to find and another solution in that case.

konstin added the needs-mre Needs more information for reproduction label Oct 8, 2024

jfcherng mentioned this issue Oct 11, 2024

uv pip install failed on Windows: stream did not contain valid UTF-8 #7549

Closed

kahojyun mentioned this issue Oct 11, 2024

Fix stream did not contain valid UTF-8 #8120

Merged

charliermarsh mentioned this issue Oct 11, 2024

I don't know why and how , clearly pip install jittor is normal #8084

Closed

konstin added bug Something isn't working and removed needs-mre Needs more information for reproduction labels Oct 12, 2024

konstin closed this as completed Oct 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Error]stream did not contain valid UTF-8 #8009

[Error]stream did not contain valid UTF-8 #8009

sdbds commented Oct 8, 2024

BurntSushi commented Oct 8, 2024

konstin commented Oct 8, 2024 •

edited

Loading

sdbds commented Oct 8, 2024 •

edited

Loading

sdbds commented Oct 8, 2024

kahojyun commented Oct 9, 2024

sdbds commented Oct 9, 2024

kahojyun commented Oct 11, 2024

BurntSushi commented Oct 11, 2024

Super1Windcloud commented Oct 11, 2024

sdbds commented Oct 12, 2024

konstin commented Oct 12, 2024

[Error]stream did not contain valid UTF-8 #8009

[Error]stream did not contain valid UTF-8 #8009

Comments

sdbds commented Oct 8, 2024

BurntSushi commented Oct 8, 2024

konstin commented Oct 8, 2024 • edited Loading

sdbds commented Oct 8, 2024 • edited Loading

sdbds commented Oct 8, 2024

kahojyun commented Oct 9, 2024

sdbds commented Oct 9, 2024

kahojyun commented Oct 11, 2024

BurntSushi commented Oct 11, 2024

Super1Windcloud commented Oct 11, 2024

very nice , how do you find this reason ?

sdbds commented Oct 12, 2024

konstin commented Oct 12, 2024

konstin commented Oct 8, 2024 •

edited

Loading

sdbds commented Oct 8, 2024 •

edited

Loading