Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python Error: 'utf-8' codec can't decode byte #24

Closed
Anomalyzero opened this issue Jul 19, 2019 · 12 comments
Closed

Python Error: 'utf-8' codec can't decode byte #24

Anomalyzero opened this issue Jul 19, 2019 · 12 comments

Comments

@Anomalyzero
Copy link

I encounter the following error when attempting to run protontricks:

user@host:~/.steam/steam/steamapps/common/They Are Billions$ protontricks 644930 -q --force vcrun2017 dotnet472 gdiplus
Traceback (most recent call last):
  File "/usr/local/bin/protontricks", line 11, in <module>
    load_entry_point('protontricks==1.2.3', 'console_scripts', 'protontricks')()
  File "/usr/local/lib/python3.6/dist-packages/protontricks/cli.py", line 227, in main
    steam_path=steam_path, steam_apps=steam_apps, appid=args.appid)
  File "/usr/local/lib/python3.6/dist-packages/protontricks/steam.py", line 444, in find_proton_app
    steam_path=steam_path, steam_apps=steam_apps, appid=appid)
  File "/usr/local/lib/python3.6/dist-packages/protontricks/steam.py", line 402, in find_steam_proton_app
    proton_appid = get_proton_appid(compat_tool_name, appinfo_path)
  File "/usr/local/lib/python3.6/dist-packages/protontricks/steam.py", line 287, in get_proton_appid
    vdf_sections = get_appinfo_sections(appinfo_path)
  File "/usr/local/lib/python3.6/dist-packages/protontricks/steam.py", line 272, in get_appinfo_sections
    vdf_d = vdf.binary_loads(data[i:i+vdf_section_size])
  File "/usr/local/lib/python3.6/dist-packages/vdf/__init__.py", line 337, in binary_loads
    stack[-1][key], idx = read_string(s, idx)
  File "/usr/local/lib/python3.6/dist-packages/vdf/__init__.py", line 305, in read_string
    result = result.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 12: invalid start byte

This is on a fresh install of python3, pip and protontools. I installed protontools by following the instructions in the README. I tried removing the python packages and reinstalling from scratch, I tried updating everything on my machine, both with pipx and standard pip, no luck.

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

Python 3.6.8
pip 9.0.1
Matoking added a commit that referenced this issue Jul 19, 2019
@Matoking
Require vdf 3.1. Refs #24
216feea 
Require version 3.1 of vdf library.
Matoking added a commit that referenced this issue Jul 19, 2019
@Matoking
Require vdf 3.1. Refs #24
c76e814 
Require version 3.1 of vdf library. Might fix parsing issue.
@Matoking
Copy link
Owner

I pushed a potential fix to vdf_update branch. If you installed protontricks using pipx, update your protontricks installation with the following command

pipx install --force --spec git+https://github.com/Matoking/protontricks.git@vdf_update protontricks

and test if it solves the issue.

@Anomalyzero
Copy link
Author

Anomalyzero commented Jul 22, 2019 •
edited
Loading

@Matoking, I ran the command, restarted the terminal, error remains unchanged:

user@host:~$ protontricks 644930 -q --force vcrun2017 dotnet472 gdiplus
Traceback (most recent call last):
  File "/home/user/.local/bin/protontricks", line 11, in <module>
    load_entry_point('protontricks==1.2.3+2.gc76e814', 'console_scripts', 'protontricks')()
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/cli.py", line 227, in main
    steam_path=steam_path, steam_apps=steam_apps, appid=args.appid)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 444, in find_proton_app
    steam_path=steam_path, steam_apps=steam_apps, appid=appid)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 402, in find_steam_proton_app
    proton_appid = get_proton_appid(compat_tool_name, appinfo_path)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 287, in get_proton_appid
    vdf_sections = get_appinfo_sections(appinfo_path)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 272, in get_appinfo_sections
    vdf_d = vdf.binary_loads(data[i:i+vdf_section_size])
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/vdf/__init__.py", line 337, in binary_loads
    stack[-1][key], idx = read_string(s, idx)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/vdf/__init__.py", line 305, in read_string
    result = result.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 12: invalid start byte

@cyro666
Copy link

cyro666 commented Jul 22, 2019 •
edited
Loading

I'm getting this same error while trying to run commands like: protontricks -c "pwd" 2100 and protontricks 2100 galliumnine. I have installed protontricks through AUR on Arch Linux. Tried both protontricks-git and protontricks packages.

EDIT: Typo

@cyro666
Copy link

cyro666 commented Jul 22, 2019 •
edited
Loading

This was the output in my case:

Traceback (most recent call last):
  File "/usr/bin/protontricks", line 11, in <module>
    load_entry_point('protontricks==1.2.3+1.gb990975', 'console_scripts', 'protontricks')()
  File "/usr/lib/python3.7/site-packages/protontricks/cli.py", line 227, in main
    steam_path=steam_path, steam_apps=steam_apps, appid=args.appid)
  File "/usr/lib/python3.7/site-packages/protontricks/steam.py", line 444, in find_proton_app
    steam_path=steam_path, steam_apps=steam_apps, appid=appid)
  File "/usr/lib/python3.7/site-packages/protontricks/steam.py", line 402, in find_steam_proton_app
    proton_appid = get_proton_appid(compat_tool_name, appinfo_path)
  File "/usr/lib/python3.7/site-packages/protontricks/steam.py", line 287, in get_proton_appid
    vdf_sections = get_appinfo_sections(appinfo_path)
  File "/usr/lib/python3.7/site-packages/protontricks/steam.py", line 272, in get_appinfo_sections
    vdf_d = vdf.binary_loads(data[i:i+vdf_section_size])
  File "/usr/lib/python3.7/site-packages/vdf/__init__.py", line 332, in binary_loads
    stack[-1][key], idx = read_string(s, idx)
  File "/usr/lib/python3.7/site-packages/vdf/__init__.py", line 305, in read_string
    result = result.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 12: invalid start byte

EDIT: Formatting

@Matoking
Copy link
Owner

@Anomalyzero @cyro666 Thanks for the reports. Seems like the "proper" appinfo.vdf parser introduced in 1.2.3 was a dud, since it's getting caught up in some of these files.

Could you make a copy of the file in ~/.steam/root/appcache/appinfo.vdf and send it to my email address (found at my GitHub profile)? That file seems to only contain publicly available information about games that the Steam client has seen at one point or another, but it doesn't hurt to be sure.

@Anomalyzero
Copy link
Author

@Matoking, file has been sent

@Matoking
Copy link
Owner

@Anomalyzero Thanks for sending me the appinfo.vdf file. I've managed to reproduce the bug.

It seems that some entries in appinfo.vdf can contain invalid UTF-8 strings. Sites like SteamDB seem to ignore the faulty characters, while the vdf Python library we're using crashes trying to parse those strings. I've submitted an upstream bug report at ValvePython/vdf#20.

For now, we can work around this by simply skipping the VDF entries that we can't parse, since we're only interested in the SteamPlay manifests. I've updated the vdf_update branch, so please run the following command again and check if Protontricks works now:

pipx install --force --spec git+https://github.com/Matoking/protontricks.git@vdf_update protontricks

@Anomalyzero
Copy link
Author

I'm afraid I'm still not seeing any difference:

user@host:~$ pipx install --force --spec git+https://github.com/Matoking/protontricks.git@vdf_update protontricks
Installing to existing directory '/home/user/.local/pipx/venvs/protontricks'
  installed package protontricks 1.2.3+2.gc76e814, Python 3.6.8
  These binaries are now globally available
    - protontricks
done! ✨ 🌟 ✨
user@host:~$ protontricks 644930 -q --force vcrun2017 dotnet472 gdiplus
Traceback (most recent call last):
  File "/home/user/.local/bin/protontricks", line 11, in <module>
    load_entry_point('protontricks==1.2.3+2.gc76e814', 'console_scripts', 'protontricks')()
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/cli.py", line 227, in main
    steam_path=steam_path, steam_apps=steam_apps, appid=args.appid)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 444, in find_proton_app
    steam_path=steam_path, steam_apps=steam_apps, appid=appid)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 402, in find_steam_proton_app
    proton_appid = get_proton_appid(compat_tool_name, appinfo_path)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 287, in get_proton_appid
    vdf_sections = get_appinfo_sections(appinfo_path)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/protontricks/steam.py", line 272, in get_appinfo_sections
    vdf_d = vdf.binary_loads(data[i:i+vdf_section_size])
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/vdf/__init__.py", line 337, in binary_loads
    stack[-1][key], idx = read_string(s, idx)
  File "/home/user/.local/pipx/venvs/protontricks/lib/python3.6/site-packages/vdf/__init__.py", line 305, in read_string
    result = result.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 12: invalid start byte

@Anomalyzero
Copy link
Author

Anomalyzero commented Jul 25, 2019 •
edited
Loading

I got it! I had to uninstall protontricks first, then run the force install command.

pipx uninstall protontricks

pipx install --force --spec git+https://github.com/Matoking/protontricks.git@vdf_update protontricks

protontricks 644930 -q --force vcrun2017 dotnet472 gdiplus

I don't know if it's useful or relevant, but I do get this warning after the last command

dotnet40 install completed, but installed file /home/anomaly/.steam/steam/steamapps/compatdata/644930/pfx/dosdevices/c:/windows/Microsoft.NET/Framework/v4.0.30319/ngen.exe not found

@lucifertdark
Copy link

I had to uninstall as well, but the fix works. :) Thanks Matoking. :)

@Anomalyzero
Copy link
Author

Agreed thanks for the prompt attention.

I'm having different issues now, but I don't know if they belong here or not :/

@Matoking
Copy link
Owner

Thanks for helping solve the issue.

I've pushed the fix to master and also released a new version 1.2.4 with the bug fix in place.

@kir68k kir68k mentioned this issue Feb 14, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Matoking @cyro666 @Anomalyzero @lucifertdark