[BUG] manifests for some Steam apps are now dicts, not plain IDs, causing CDNClient.get_manifests to crash #436

smcv · 2023-05-05T16:17:14Z

Description
Some Steam apps (the one I'm aware of is 1628350 "Steam Linux Runtime - Sniper") have changed their ['apps'][id]['manifests'][branch] from a plain manifest ID (ASCII decimal string) to a key/value dictionary with members gid, size and download, causing CDNClient.get_manifests to crash.

Steps to Reproduce the behavior
Run https://gist.github.com/smcv/8430646f3cf5e502711ddf3ea620eaea with --interactive. You might need a Steam account that has added 1628350 "Steam Linux Runtime - Sniper" to the Steam library - it's free, and is used by Proton 8.0.

Expected behavior
The script can download and display VERSION.txt from each branch of each of the Steam Linux Runtime depots.

Screenshots
For 1391110 "Steam Linux Runtime - Soldier" my script is still working:

App ID: 1391110 (Steam Linux Runtime - Soldier)
	Branch: public
		Build ID: 10940185
		Updated: 2023-04-26T15:48:12+00:00
		Depot: 1391111
			Manifest: 5271457197581595980
				#Name	Version		Runtime	Runtime_Version	Comment
				depot	0.20230405.47174			# Overall version number
				pressure-vessel	0.20230405.0	scout		# pressure-vessel-bin.tar.gz
				scripts	0.20230405.0			# from steam-runtime-tools
				soldier	0.20230405.47174	soldier	0.20230405.47174	# soldier_platform_0.20230405.47174/

but for 1628350 "Steam Linux Runtime - Sniper" it now crashes:

App ID: 1628350 (Steam Linux Runtime - Sniper)
	Branch: public
		Build ID: 10940192
		Updated: 2023-04-26T15:50:07+00:00
		Depot: 1628351
			Manifest: {'gid': '820291660013951810', 'size': '703682881', 'download': '188124800'}
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "REDACTED/python-steam/steam/client/cdn.py", line 785, in async_fetch_manifest
    app_id, depot_id, int(manifest_gid), branch_name, branch_pass
                      ^^^^^^^^^^^^^^^^^
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'dict'
2023-05-05T16:09:39Z <Greenlet at 0x7f0665bee200: async_fetch_manifest(1628350, 1628351, {'gid': '820291660013951810', 'size': '703682881',, True, 1628351, branch_name='public', branch_pass=None)> failed with TypeError

Traceback (most recent call last):
  File "REDACTED/python-steam/./describe-depots.py", line 204, in <module>
    main()
  File "REDACTED/python-steam/./describe-depots.py", line 183, in main
    Main(**vars(args)).run()
  File "REDACTED/python-steam/./describe-depots.py", line 158, in run
    for manifest in cdn.get_manifests(
  File "REDACTED/python-steam/steam/client/cdn.py", line 857, in get_manifests
    result = task.get()
  File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/usr/lib/python3/dist-packages/gevent/_compat.py", line 66, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "REDACTED/python-steam/steam/client/cdn.py", line 785, in async_fetch_manifest
    app_id, depot_id, int(manifest_gid), branch_name, branch_pass
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'dict'

Versions Report

python -m steam.versions_report

(Run python -m steam.versions_report and paste the output below)

steam: 1.4.4

Dependencies:
                 vdf: 3.4
            protobuf: 4.21.12
            requests: 2.28.1
          cachetools: 5.2.0
              gevent: 22.10.2
 gevent-eventemitter: Not Installed
       pycryptodomex: 3.11.0
              enum34: Not Installed
       win-inet-pton: Not Installed

Python runtime:
          executable: /usr/bin/python3
             version: 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
            platform: linux

System info:
              system: Linux
             machine: x86_64
             release: 6.1.0-7-amd64
             version: #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-1 (2023-03-19)

The text was updated successfully, but these errors were encountered:

smcv · 2023-05-05T16:24:26Z

This appears to be caused by the VDF returned for EMsg.ClientPICSProductInfoRequest having changed. For soldier I get

	"depots"
	{
		"1391111"
		{
			"config"
			{
				"oslist"		"linux"
				"osarch"		"64"
				"language"		""
			}
			"manifests"
			{
				"public"		"5271457197581595980"
				"client_beta"		"3684425101439344717"
				"previous_release"		"1306141847442826428"
			}
...

but for sniper I now get

	"depots"
	{
		"1628351"
		{
			"config"
			{
				"oslist"		"linux"
				"osarch"		"64"
				"language"		""
			}
			"manifests"
			{
				"public"
				{
					"gid"		"820291660013951810"
					"size"		"703682881"
					"download"		"188124800"
				}
				"client_beta"
				{
...

Some depots now have a dict like {"public": {"gid": GID, ...}, ...} where we would traditionally have seen a flat dict {"public": GID, ...}. Not all depots have this: at the time of writing, 1628350 "Steam Linux Runtime - Sniper" does, but 1391110 "Steam Linux Runtime - Soldier" does not. Presumably this is a change that is being rolled out gradually to let the Steam client give better estimates of the size of a depot. Cope with either format, by retrieving the gid field if the nested dict is present. Resolves: ValvePython#436 Signed-off-by: Simon McVittie <smcv@collabora.com>

smcv · 2023-05-05T16:40:05Z

Fix proposed in #437.

rossengeorgiev · 2023-05-05T23:30:08Z

Can confirm that there a changes to product info data. I'll need to review all parts that are affected in the code base.

DevilXD · 2023-06-11T16:12:36Z

This issue is breaking my use-case. Is there any ETA on when will this be merged?

App ID I'm using: 596350.

smcv · 2023-06-13T15:51:11Z

This issue is breaking my use-case

Does #437 solve this for the subset of the API you use? If yes, perhaps the maintainer could consider merging that as a stopgap solution, even if there are other locations needing a similar change?

Totengeist · 2023-06-13T16:30:54Z

Does #437 solve this for the subset of the API you use?

It works fine for my very simplistic use-case.

DevilXD · 2023-06-13T16:58:07Z

@smcv Yes, I'm actually already using that PR instead of the official package while I'm waiting, and it does work. Haven't encountered any issues. My use-case involves downloading a single file from via the CDN, from a particular app ID and depot, latest manifest.

mommalongnips · 2023-07-23T23:21:19Z

My use case is also suffering from this issue. @smcv your fix resolves it for me. Can we merge in this change? Please!

MichaelHinrichs · 2023-11-10T15:52:19Z

Using file_list = mycdn.iter_files(appID) works with @smcv's solution, but now i can't use next(file_list), print(file_list), print(my_file) one at a time, or list(file_list)[:len(file_list)]. I just get the same error from the beginning of this issue.

Some depots now have a dict like {"public": {"gid": GID, ...}, ...} where we would traditionally have seen a flat dict {"public": GID, ...}. Not all depots have this: at the time of writing, 1628350 "Steam Linux Runtime - Sniper" does, but 1391110 "Steam Linux Runtime - Soldier" does not. Presumably this is a change that is being rolled out gradually to let the Steam client give better estimates of the size of a depot. Cope with either format, by retrieving the gid field if the nested dict is present. Resolves: ValvePython#436 Signed-off-by: Simon McVittie <smcv@collabora.com> (cherry picked from commit 783f023)

smcv added bug needs-review labels May 5, 2023

smcv changed the title ~~[BUG]~~ [BUG] manifests for some Steam apps are now dicts, not plain IDs, causing CDNClient.get_manifests to crash May 5, 2023

smcv linked a pull request May 5, 2023 that will close this issue

CDNClient: Cope with longer manifest descriptions for some depots #437

Open

Gobot1234 added a commit to Gobot1234/steam.py that referenced this issue May 5, 2023

Fix for ValvePython/steam#436

2292ca9

rossengeorgiev removed the needs-review label May 5, 2023

fredi-68 mentioned this issue Jan 22, 2024

TypeError: int() argument must be a string, a bytes-like object or a real number, not 'dict' fredi-68/steam-disable-update#6

Open

StalkR mentioned this issue Apr 25, 2024

client/cdn: manifest returns dict, gid is a key #461

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] manifests for some Steam apps are now dicts, not plain IDs, causing CDNClient.get_manifests to crash #436

[BUG] manifests for some Steam apps are now dicts, not plain IDs, causing CDNClient.get_manifests to crash #436

smcv commented May 5, 2023

smcv commented May 5, 2023

smcv commented May 5, 2023

rossengeorgiev commented May 5, 2023

DevilXD commented Jun 11, 2023

smcv commented Jun 13, 2023

Totengeist commented Jun 13, 2023

DevilXD commented Jun 13, 2023

mommalongnips commented Jul 23, 2023

MichaelHinrichs commented Nov 10, 2023

[BUG] manifests for some Steam apps are now dicts, not plain IDs, causing CDNClient.get_manifests to crash #436

[BUG] manifests for some Steam apps are now dicts, not plain IDs, causing CDNClient.get_manifests to crash #436

Comments

smcv commented May 5, 2023

smcv commented May 5, 2023

smcv commented May 5, 2023

rossengeorgiev commented May 5, 2023

DevilXD commented Jun 11, 2023

smcv commented Jun 13, 2023

Totengeist commented Jun 13, 2023

DevilXD commented Jun 13, 2023

mommalongnips commented Jul 23, 2023

MichaelHinrichs commented Nov 10, 2023